24 research outputs found
SupeRVol: Super-Resolution Shape and Reflectance Estimation in Inverse Volume Rendering
We propose an end-to-end inverse rendering pipeline called SupeRVol that
allows us to recover 3D shape and material parameters from a set of color
images in a super-resolution manner. To this end, we represent both the
bidirectional reflectance distribution function (BRDF) and the signed distance
function (SDF) by multi-layer perceptrons. In order to obtain both the surface
shape and its reflectance properties, we revert to a differentiable volume
renderer with a physically based illumination model that allows us to decouple
reflectance and lighting. This physical model takes into account the effect of
the camera's point spread function thereby enabling a reconstruction of shape
and material in a super-resolution quality. Experimental validation confirms
that SupeRVol achieves state of the art performance in terms of inverse
rendering quality. It generates reconstructions that are sharper than the
individual input images, making this method ideally suited for 3D modeling from
low-resolution imagery
An Introduction to Optimization Techniques in Computer Graphics
International audienceBackground: Many students in Computer Science do not have a sufficient background in applied mathematics to employ state-of-the-art optimization techniques and to judge the outcome of such techniques critically (e.g. regarding the stability/quality/accuracy of their output). At the same time, the use of optimization techniques in computer graphics is becoming ubiquitous. Treating optimization algorithms as a black box yields sub-optimal results at best. At worst, stability issues and convergence problems may prevent the solution of a problem or impede the general application of a method to a wide range of input, i.e. beyond the set of examples shown in a paper. The course will draw attention to these aspects and to current best practices. This will enable participants to judge articles that use optimization schemes critically and improve their own skill set
3D-MuPPET: 3D Multi-Pigeon Pose Estimation and Tracking
Markerless methods for animal posture tracking have been developing recently,
but frameworks and benchmarks for tracking large animal groups in 3D are still
lacking. To overcome this gap in the literature, we present 3D-MuPPET, a
framework to estimate and track 3D poses of up to 10 pigeons at interactive
speed using multiple-views. We train a pose estimator to infer 2D keypoints and
bounding boxes of multiple pigeons, then triangulate the keypoints to 3D. For
correspondence matching, we first dynamically match 2D detections to global
identities in the first frame, then use a 2D tracker to maintain
correspondences accross views in subsequent frames. We achieve comparable
accuracy to a state of the art 3D pose estimator for Root Mean Square Error
(RMSE) and Percentage of Correct Keypoints (PCK). We also showcase a novel use
case where our model trained with data of single pigeons provides comparable
results on data containing multiple pigeons. This can simplify the domain shift
to new species because annotating single animal data is less labour intensive
than multi-animal data. Additionally, we benchmark the inference speed of
3D-MuPPET, with up to 10 fps in 2D and 1.5 fps in 3D, and perform quantitative
tracking evaluation, which yields encouraging results. Finally, we show that
3D-MuPPET also works in natural environments without model fine-tuning on
additional annotations. To the best of our knowledge we are the first to
present a framework for 2D/3D posture and trajectory tracking that works in
both indoor and outdoor environments
Introducing total curvature for image processing
We introduce the novel continuous regularizer total curvature (TC) for images u: Ω → R. It is defined as the Menger-Melnikov curvature of the Radon measure |Du|, which can be understood as a measure theoretic formulation of curvature mathematically related to mean curvature. The functional is not convex, therefore we define a convex relaxation which yields a close approximation. Similar to the total variation, the relaxation can be written as the support functional of a convex set, which means that there are stable and efficient minimization algorithms available when it is used as a regularizer in image processing problems. Our current implementation can handle general inverse problems, inpainting and segmentation. We demonstrate in experiments and comparisons how the regularizer performs in practice. 1
SPACETIME-CONTINUOUS GEOMETRY MESHES FROM MULTI-VIEW VIDEO SEQUENCES
We reconstruct geometry for a time-varying scene given by a number of video sequences. The dynamic geometry is represented by a 3D hypersurface embedded in space-time. The intersection of the hypersurface with planes of constant time then yields the geometry at a single time instant. In this paper, we model the hypersurface with a collection of triangle meshes, one for each time frame. The photo-consistency error is measured by an error functional defined as an integral over the hypersurface. It can be minimized using a PDE driven surface evolution, which simultaneously optimizes space-time continuity as well. Compared to our previous implementation based on level sets, triangle meshes yield more accurate results, while requiring less memory and computation time. Meshes are also directly compatible with triangle-based rendering algorithms, so no additional post-processing is required. 1
An approach to vectorial total variation based on geometric measure theory
We analyze a previously unexplored generalization of the scalar total variation to vector-valued functions, which is motivated by geometric measure theory. A complete mathematical characterization is given, which proves important invariance properties as well as existence of solutions of the vectorial ROF model. As an important feature, there exists a dual formulation for the proposed vectorial total variation, which leads to a fast and stable minimization algorithm. The main difference to previous approaches with similar properties is that we penalize across a common edge direction for all channels, which is a major theoretical advantage. Experiments show that this leads to a significiantly better restoration of color edges in practice. 1
Space-Time Isosurface Evolution for Temporally Coherent 3D Reconstruction
We model the dynamic geometry of a time-varying scene as a 3D isosurface in
space-time.
The intersection of the isosurface with planes of constant time yields the
geometry at a
single time instant.
An optimal fit of our model to multiple video sequences is
defined as the minimum of an energy functional.
This functional is given by an integral over the entire hypersurface,
which is designed to optimize photo-consistency.
A PDE-based evolution derived from the Euler-Lagrange equation
maximizes consistency with all of the given video data simultaneously.
The result is a 3D model of the scene which varies smoothly over time.
The geometry reconstructed by this scheme
is significantly better than results obtained by space-carving approaches
that do not enforce temporal coherence
Superresolution texture maps for multiview reconstruction
We study the scenario of a multiview setting, where several calibrated views of a textured object with known surface geometry are available. The objective is to estimate a diffuse texture map as precisely as possible. A superresolution image formation model based on the camera properties leads to a total variation energy for the desired texture map, which can be recovered as the minimizer of the functional by solving the Euler-Lagrange equation on the surface. The PDE is transformed to planar texture space via an automatically created conformal atlas, where it can be solved using total variation deblurring. The proposed approach allows to recover a high-resolution, high-quality texture map even from lower-resolution photographs, which is of interest for a variety of image-based modeling applications. 1
Reconstructing the geometry of flowing water
We present a recording scheme, image formation model and reconstruction method that enables image-based modeling of flowing bodies of water from multi-video input data. The recorded water is dyed with a fluorescent chemical to measure the thickness of a column of water, which leads to an image formation model based on integrated emissivities along a viewing ray. This model allows for a photoconsistency based error measure for a weighted minimal surface, which is recovered using a PDE obtained from the Euler-Lagrangian formulation of the problem. The resulting equation is solved using the level set method. 1